{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wide and Deep on TensorFlow (notebook style)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2016 Google Inc. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
    " http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This notebook uses the tf.learn API in TensorFlow to answer a yes/no question. This is called a binary classification problem（二分类问题）: Given census data about a person such as age, gender, education and occupation (the features), we will try to predict whether or not the person earns more than 50,000 dollars a year (the target label). \n",
    "\n",
    "Given an individual's information our model will output a number between 0 and 1, which can be interpreted（说明） as the model's certainty that the individual has an annual income of over 50,000 dollars, (1=True, 0=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports and constants（常量）\n",
    "First we'll import our libraries and set up some strings for column names. We also print out the version of TensorFlow we are running.\n",
    "\n",
    "__future__是py2的概念，对应py2，py3就是future，这是为了在是py2的时候还能用到一些新版本的特性而做成的包。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using TensorFlow version 1.9.0\n",
      "\n",
      "Feature columns are:  ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13', 'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'] \n",
      "\n",
      "Columns and data as a dict:  {'I1': 0, 'I2': 127, 'I3': 1, 'I4': 3, 'I5': 1683, 'I6': 19, 'I7': 26, 'I8': 17, 'I9': 475, 'I10': 0, 'I11': 9, 'I12': 0, 'I13': 3, 'C1': '05db9164', 'C2': '8947f767', 'C3': '11c9d79e', 'C4': '52a787c8', 'C5': '4cf72387', 'C6': 'fbad5c96', 'C7': '18671b18', 'C8': '0b153874', 'C9': 'a73ee510', 'C10': 'ceb10289', 'C11': '77212bd7', 'C12': '79507c6b', 'C13': '7203f04e', 'C14': '07d13a8f', 'C15': '2c14c412', 'C16': '49013ffe', 'C17': '8efede7f', 'C18': 'bd17c3da', 'C19': 'f6a3e43b', 'C20': 'a458ea53', 'C21': '35cd95c9', 'C22': 'ad3062eb', 'C23': 'c7dc6720', 'C24': '3fdb382b', 'C25': '010f6491', 'C26': '49d68486'} \n",
      "\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import time\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO) # Set to INFO for tracking training, default is WARN. ERROR for least messages\n",
    "\n",
    "print(\"Using TensorFlow version %s\\n\" % (tf.__version__))\n",
    "\n",
    "\n",
    "CONTINUOUS_COLUMNS =  [\"I\"+str(i) for i in range(1,14)] # 1-13 inclusive\n",
    "CATEGORICAL_COLUMNS = [\"C\"+str(i) for i in range(1,27)] # 1-26 inclusive\n",
    "LABEL_COLUMN = [\"clicked\"]\n",
    "\n",
    "TRAIN_DATA_COLUMNS = LABEL_COLUMN + CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "# TEST_DATA_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "\n",
    "FEATURE_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "\n",
    "print('Feature columns are: ', FEATURE_COLUMNS, '\\n')\n",
    "\n",
    " # label is 1\n",
    "sample = [ 0 , 2, 11, 5, 10262, 34, 2, 4, 5,0 , 1,0 , 5, \"be589b51\", \"287130e0\", \"cd7a7a22\", \"fb7334df\", \"25c83c98\",\"0\" , \"6cdb3998\", \"361384ce\", \"a73ee510\", \"3ff10fb2\", \"5874c9c9\", \"976cbd4c\", \"740c210d\", \"1adce6ef\", \"310d155b\", \"07eb8110\", \"07c540c4\", \"891589e7\", \"18259a83\", \"a458ea53\", \"a0ab60ca\",\"0\" , \"32c7478e\", \"a052b1ed\", \"9b3e8820\", \"8967c0d2\"]\n",
    "\n",
    "# label is 1\n",
    "sample = [ 0, 127, 1, 3, 1683, 19, 26, 17, 475, 0, 9, 0, 3, \"05db9164\", \"8947f767\", \"11c9d79e\", \"52a787c8\", \"4cf72387\", \"fbad5c96\", \"18671b18\", \"0b153874\", \"a73ee510\", \"ceb10289\", \"77212bd7\", \"79507c6b\", \"7203f04e\", \"07d13a8f\", \"2c14c412\", \"49013ffe\", \"8efede7f\", \"bd17c3da\", \"f6a3e43b\", \"a458ea53\", \"35cd95c9\", \"ad3062eb\", \"c7dc6720\", \"3fdb382b\", \"010f6491\", \"49d68486\"]\n",
    "\n",
    "print('Columns and data as a dict: ', dict(zip(FEATURE_COLUMNS, sample)), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input file parsing（分析）\n",
    "\n",
    "This section puts the file into a `Reader` which reads from the file one batch at a time. \n",
    "\n",
    "We set up the Tensors（张量） to be a dictionary of features mapping from their string name to the tensor value.\n",
    "\n",
    "Note that the `_input_fn()` function is wrapped, enabling it to be used for different files.\n",
    "\n",
    "NOTE: This reads from the input file directly via TensorFlow, rather than using an intermediate tool such as pandas to load the entire dataset into memory first. This is done to enable the system to scale to large inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More about input functions\n",
    "\n",
    "The input function is how we will feed the input data into the model during training and evaluation. \n",
    "The structure that must be returned is a pair, where the first element is a dict of the column names (features) mapped to a tensor of values, and the 2nd element is a tensor of values representing the answers (labels). Recall that a tensor is just a general term for an n-dimensional array（n维数组）.\n",
    "\n",
    "This could be represented as: `map(column_name => [Tensor of values]) , [Tensor of labels])`\n",
    "\n",
    "More concretely, for this particular dataset, something like this:\n",
    "\n",
    "    { \n",
    "      'age':            [ 39, 50, 38, 53, 28, … ], \n",
    "      'marital_status': [ 'Married-civ-spouse', 'Never-married', 'Widowed', 'Widowed' … ],\n",
    "       ...\n",
    "      'gender':           ['Male', 'Female', 'Male', 'Male', 'Female',, … ], \n",
    "    } , \n",
    "    [ 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1]\n",
    "    \n",
    "Additionally, we define which columns of the input data we will treat as categorical（无序类别） vs continuous（连续）, using the global `CATEGORICAL_COLUMNS`.\n",
    "\n",
    "You can try different values for `BATCH_SIZE` to see how they impact your results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### High-level structure of input functions for CSV-style data\n",
    "1. Queue file(s) （队列文件）\n",
    "2. Read a batch of data from the next file\n",
    "3. Create record defaults, generally 0 for continuous values, and \"\" for categorical. You can use named types if you prefer\n",
    "4. Decode the CSV and restructure it to be appropriate for the graph's input format\n",
    "    * `zip()` column headers with the data\n",
    "    * `pop()` off the label column(s)（移除元素）\n",
    "    * Remove/pop any unneeded column(s)\n",
    "    * Run `tf.expand_dims()` on categorical columns\n",
    "    5. Return the pair: `(feature_dict, label_array)`\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input function configured\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 400\n",
    "\n",
    "def generate_input_fn(filename, batch_size=BATCH_SIZE):\n",
    "    def _input_fn():\n",
    "        filename_queue = tf.train.string_input_producer([filename])\n",
    "        reader = tf.TextLineReader()\n",
    "        # Reads out batch_size number of lines\n",
    "        key, value = reader.read_up_to(filename_queue, num_records=batch_size)\n",
    "        \n",
    "        # 1 int label, 13 ints, 26 strings\n",
    "        cont_defaults = [ [0] for i in range(1,14) ]\n",
    "        cate_defaults = [ [\" \"] for i in range(1,27) ]\n",
    "        label_defaults = [ [0] ]\n",
    "        column_headers = TRAIN_DATA_COLUMNS\n",
    "        # The label is the first column of the data.\n",
    "        record_defaults = label_defaults + cont_defaults + cate_defaults\n",
    "\n",
    "        # Decode CSV data that was just read out. \n",
    "        # Note that this does NOT return a dict, \n",
    "        # so we will need to zip it up with our headers\n",
    "        columns = tf.decode_csv(\n",
    "            value, record_defaults=record_defaults)\n",
    "        \n",
    "        # all_columns is a dictionary that maps from column names to tensors of the data.\n",
    "        all_columns = dict(zip(column_headers, columns))\n",
    "        \n",
    "        # Pop and save our labels \n",
    "        # dict.pop() returns the popped array of values; exactly what we need!\n",
    "        labels = all_columns.pop(LABEL_COLUMN[0])\n",
    "        \n",
    "        # the remaining columns are our features\n",
    "        features = all_columns \n",
    "\n",
    "        # Sparse categorical features must be represented with an additional dimension. \n",
    "        # There is no additional work needed for the Continuous columns; they are the unaltered columns.\n",
    "        # See docs for tf.SparseTensor for more info\n",
    "        for feature_name in CATEGORICAL_COLUMNS:\n",
    "            features[feature_name] = tf.expand_dims(features[feature_name], -1)\n",
    "\n",
    "        return features, labels\n",
    "\n",
    "    return _input_fn\n",
    "\n",
    "print('input function configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Feature Columns\n",
    "This section configures the model with the information about the model. There are many parameters here to experiment with to see how they affect the accuracy.\n",
    "\n",
    "This is the bulk of the time and energy that is often spent on making a machine learning model work, called *feature selection* or *feature engineering*. We choose the features (columns) we will use for training, and apply any additional transformations to them as needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sparse Columns（稀疏列）\n",
    "First we build the sparse columns.\n",
    "\n",
    "Use `sparse_column_with_keys()` for columns that we know all possible values for.\n",
    "\n",
    "Use `sparse_column_with_hash_bucket()` for columns that we want the the library to automatically map values for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wide/Sparse columns configured\n"
     ]
    }
   ],
   "source": [
    "# Sparse base columns.\n",
    "# C1 = tf.contrib.layers.sparse_column_with_hash_bucket('C1', hash_bucket_size=1000)\n",
    "# C2 = tf.contrib.layers.sparse_column_with_hash_bucket('C2', hash_bucket_size=1000)\n",
    "# C3 = tf.contrib.layers.sparse_column_with_hash_bucket('C3', hash_bucket_size=1000)\n",
    "# ...\n",
    "# Cn = tf.contrib.layers.sparse_column_with_hash_bucket('Cn', hash_bucket_size=1000)\n",
    "# wide_columns = [C1, C2, C3, ... , Cn]\n",
    "\n",
    "wide_columns = []\n",
    "for name in CATEGORICAL_COLUMNS:\n",
    "    wide_columns.append(tf.contrib.layers.sparse_column_with_hash_bucket(\n",
    "            name, hash_bucket_size=1000))\n",
    "\n",
    "print('Wide/Sparse columns configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuous columns\n",
    "Second, configure the real-valued columns using `real_valued_column()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deep/continuous columns configured\n"
     ]
    }
   ],
   "source": [
    "# Continuous base columns.\n",
    "# I1 = tf.contrib.layers.real_valued_column(\"I1\")\n",
    "# I2 = tf.contrib.layers.real_valued_column(\"I2\")\n",
    "# I3 = tf.contrib.layers.real_valued_column(\"I3\")\n",
    "# ...\n",
    "# In = tf.contrib.layers.real_valued_column(\"In\")\n",
    "# deep_columns = [I1, I2, I3, ... , In]\n",
    "\n",
    "deep_columns = []\n",
    "for name in CONTINUOUS_COLUMNS:\n",
    "    deep_columns.append(tf.contrib.layers.real_valued_column(name))\n",
    "\n",
    "print('deep/continuous columns configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformations\n",
    "Now for the interesting stuff. We will employ a couple of techniques to get even more out of the data.\n",
    " \n",
    "* **bucketizing** turns what would have otherwise been a continuous feature into a categorical one. \n",
    "* **feature crossing** allows us to compute a model weight for specific pairings across columns, rather than learning them as independently. This essentially encodes related columns together, for situations where having 2 (or more) columns being certain values is meaningful. \n",
    "\n",
    "Only categorical features can be crossed. This is one reason why age has been bucketized.\n",
    "\n",
    "For example, crossing education and occupation would enable the model to learn about: \n",
    "\n",
    "    education=\"Bachelors\" AND occupation=\"Exec-managerial\"\n",
    "\n",
    "or perhaps \n",
    "\n",
    "    education=\"Bachelors\" AND occupation=\"Craft-repair\"\n",
    "\n",
    "We do a few combined features (feature crosses) here. \n",
    "\n",
    "Add your own, based on your intuitions about the dataset, to try to improve on the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transformations complete\n"
     ]
    }
   ],
   "source": [
    "# No known Transformations. Can add some if desired. \n",
    "# Examples from other datasets are shown below.\n",
    "\n",
    "# age_buckets = tf.contrib.layers.bucketized_column(age,\n",
    "#             boundaries=[ 18, 25, 30, 35, 40, 45, 50, 55, 60, 65 ])\n",
    "# education_occupation = tf.contrib.layers.crossed_column([education, occupation], \n",
    "#                                                         hash_bucket_size=int(1e4))\n",
    "# age_race_occupation = tf.contrib.layers.crossed_column([age_buckets, race, occupation], \n",
    "#                                                        hash_bucket_size=int(1e6))\n",
    "# country_occupation = tf.contrib.layers.crossed_column([native_country, occupation], \n",
    "#                                                       hash_bucket_size=int(1e4))\n",
    "\n",
    "print('Transformations complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group feature columns into 2 objects\n",
    "\n",
    "The wide columns are the sparse（稀疏的）, categorical columns that we specified, as well as our hashed, bucket, and feature crossed columns. \n",
    "\n",
    "The deep columns are composed of embedded categorical columns along with the continuous real-valued columns. **Column embeddings** transform a sparse, categorical tensor into a low-dimensional and dense real-valued vector. The embedding values are also trained along with the rest of the model. For more information about embeddings, see the TensorFlow tutorial on [Vector Representations Words](https://www.tensorflow.org/tutorials/word2vec/), or [Word Embedding](https://en.wikipedia.org/wiki/Word_embedding) on Wikipedia.\n",
    "\n",
    "The higher the dimension of the embedding is, the more degrees of freedom the model will have to learn the representations of the features. We are starting with an 8-dimension embedding for simplicity, but later you can come back and increase the dimensionality if you wish.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "wide and deep columns configured\n"
     ]
    }
   ],
   "source": [
    "# Wide columns and deep columns.\n",
    "# wide_columns = [gender, race, native_country,\n",
    "#       education, occupation, workclass,\n",
    "#       marital_status, relationship,\n",
    "#       age_buckets, education_occupation,\n",
    "#       age_race_occupation, country_occupation]\n",
    "\n",
    "# deep_columns = [\n",
    "#   tf.contrib.layers.embedding_column(workclass, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(education, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(marital_status, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(gender, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(relationship, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(race, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(native_country, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(occupation, dimension=8),\n",
    "#   age,\n",
    "#   education_num,\n",
    "#   capital_gain,\n",
    "#   capital_loss,\n",
    "#   hours_per_week,\n",
    "# ]\n",
    "\n",
    "# Embeddings for wide columns into deep columns\n",
    "for col in wide_columns:\n",
    "    deep_columns.append(tf.contrib.layers.embedding_column(col, \n",
    "                                                           dimension=8))\n",
    "\n",
    "print('wide and deep columns configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the model\n",
    "\n",
    "You can train either a \"wide\" model, a \"deep\" model, or a \"wide and deep\" model, using the classifiers below. Try each one and see what kind of results you get.\n",
    "\n",
    "* **Wide**: Linear Classifier\n",
    "* **Deep**: Deep Neural Net Classifier\n",
    "* **Wide & Deep**: Combined Linear and Deep Classifier\n",
    "\n",
    "The `hidden_units` or `dnn_hidden_units`（隐层神经元数目） argument is to specify the size of each layer of the deep portion of the network. For example, `[12, 20, 15]` would create a network with the first layer of size 12, the second layer of size 20, and a third layer of size 15."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model directory = models/model_WIDE_AND_DEEP_1559813118\n",
      "WARNING:tensorflow:From <ipython-input-7-f2cacdb53631>:15: RunConfig.__init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.\n",
      "WARNING:tensorflow:From <ipython-input-7-f2cacdb53631>:40: calling DNNLinearCombinedClassifier.__init__ (from tensorflow.contrib.learn.python.learn.estimators.dnn_linear_combined) with fix_global_step_increment_bug=False is deprecated and will be removed after 2017-04-15.\n",
      "Instructions for updating:\n",
      "Please set fix_global_step_increment_bug=True and update training steps in your pipeline. See pydoc for details.\n",
      "WARNING:tensorflow:From /Users/wuhuan/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py:676: multi_class_head (from tensorflow.contrib.learn.python.learn.estimators.head) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.contrib.estimator.*_head.\n",
      "WARNING:tensorflow:From /Users/wuhuan/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py:1179: BaseEstimator.__init__ (from tensorflow.contrib.learn.python.learn.estimators.estimator) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please replace uses of any Estimator from tf.contrib.learn with an Estimator from tf.estimator.*\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1a41b3f0b8>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 100, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_WIDE_AND_DEEP_1559813118'}\n",
      "estimator built\n"
     ]
    }
   ],
   "source": [
    "def create_model_dir(model_type):\n",
    "    # Returns something like models/model_WIDE_AND_DEEP_1493043407\n",
    "    return 'models/model_' + model_type + '_' + str(int(time.time()))\n",
    "\n",
    "# Specify the desired model_dir \n",
    "def get_model(model_type, model_dir):\n",
    "    print(\"Model directory = %s\" % model_dir)\n",
    "    \n",
    "    # There are more options here than shown here. \n",
    "    # We are using this to show additional checkpointing for illustrative purposes.\n",
    "    # In a real system with far more samples, you would \n",
    "    #     likely choose to save checkpoints less frequently.\n",
    "    runconfig = tf.contrib.learn.RunConfig(\n",
    "        save_checkpoints_secs=None,\n",
    "        save_checkpoints_steps = 100,\n",
    "    )\n",
    "    \n",
    "    m = None\n",
    "    \n",
    "    # Linear Classifier\n",
    "    if model_type == 'WIDE':\n",
    "        m = tf.contrib.learn.LinearClassifier(\n",
    "            model_dir=model_dir, \n",
    "            feature_columns=wide_columns)\n",
    "\n",
    "    # Deep Neural Net Classifier\n",
    "    if model_type == 'DEEP':\n",
    "        m = tf.contrib.learn.DNNClassifier(\n",
    "            model_dir=model_dir,\n",
    "            feature_columns=deep_columns,\n",
    "            hidden_units=[100, 50, 25])\n",
    "\n",
    "    # Combined Linear and Deep Classifier\n",
    "    if model_type == 'WIDE_AND_DEEP':\n",
    "        m = tf.contrib.learn.DNNLinearCombinedClassifier(\n",
    "            model_dir=model_dir,\n",
    "            linear_feature_columns=wide_columns,\n",
    "            dnn_feature_columns=deep_columns,\n",
    "            dnn_hidden_units=[100, 70, 50, 25],\n",
    "            config=runconfig)\n",
    "        \n",
    "    print('estimator built')\n",
    "    \n",
    "    return m\n",
    "    \n",
    "\n",
    "MODEL_TYPE = 'WIDE_AND_DEEP'\n",
    "model_dir = create_model_dir(model_type=MODEL_TYPE)\n",
    "m = get_model(model_type=MODEL_TYPE, model_dir=model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Showing that canned estimators return an instance of 'Evaluable'\n",
    "\n",
    "from tensorflow.contrib.learn.python.learn import evaluable\n",
    "isinstance(m, evaluable.Evaluable)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fit the model (train it)\n",
    "\n",
    "Run `fit()` to train the model. You can experiment with the `train_steps` and `BATCH_SIZE` parameters.\n",
    "\n",
    "This can take some time, depending on the values chosen for `train_steps` and `BATCH_SIZE`.\n",
    "\n",
    "Our datafile is hosted on Google Cloud Storage; the reader we created at the beginning knows how to read from it.\n",
    "\n",
    "If you don't want to download a new copy of the dataset each time your script runs, you can download it locally using \n",
    "\n",
    "    gsutil cp gs://dataset-uploader/criteo-kaggle/medium_version/train.csv .\n",
    "    gsutil cp gs://dataset-uploader/criteo-kaggle/medium_version/eval.csv .\n",
    "    \n",
    "If you want to download it manually, use\n",
    "\n",
    "- http://storageapis.google.com/dataset-uploader/criteo-kaggle/medium_version/eval.csv\n",
    "- http://storageapis.google.com/dataset-uploader/criteo-kaggle/medium_version/train.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the cloud or local depending on your preference\n",
    "\n",
    "# CLOUD\n",
    "#train_file = \"gs://dataset-uploader/criteo-kaggle/medium_version/train.csv\"\n",
    "#eval_file  = \"gs://dataset-uploader/criteo-kaggle/medium_version/eval.csv\"\n",
    "\n",
    "# LOCAL. Update these paths as appropriate\n",
    "train_file = \"data/train.csv\"\n",
    "eval_file  = \"data/eval.csv\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From /Users/wuhuan/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:800: calling expand_dims (from tensorflow.python.ops.array_ops) with dim is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use the `axis` argument instead\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:From /Users/wuhuan/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/head.py:678: ModelFnOps.__new__ (from tensorflow.contrib.learn.python.learn.estimators.model_fn) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "When switching to tf.estimator.Estimator, use tf.estimator.EstimatorSpec. You can use the `estimator_spec` method to create an equivalent one.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:loss = 334.3153, step = 2\n",
      "INFO:tensorflow:Saving checkpoints for 102 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.09167\n",
      "INFO:tensorflow:loss = 0.7048233, step = 202 (25.675 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 204 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 10.4961\n",
      "INFO:tensorflow:Saving checkpoints for 306 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 41.1855\n",
      "INFO:tensorflow:loss = 0.5280818, step = 402 (4.952 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 408 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.4166\n",
      "INFO:tensorflow:Saving checkpoints for 510 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 41.0878\n",
      "INFO:tensorflow:loss = 0.47872123, step = 602 (4.917 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 612 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.2876\n",
      "INFO:tensorflow:Saving checkpoints for 714 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 41.1279\n",
      "INFO:tensorflow:loss = 0.51457626, step = 802 (4.941 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 816 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.1992\n",
      "INFO:tensorflow:Saving checkpoints for 918 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 41.068\n",
      "INFO:tensorflow:loss = 0.55771697, step = 1002 (4.935 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1020 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.4964\n",
      "INFO:tensorflow:Saving checkpoints for 1122 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 39.1243\n",
      "INFO:tensorflow:loss = 0.46587813, step = 1202 (5.056 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1224 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.8286\n",
      "INFO:tensorflow:Saving checkpoints for 1326 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.545\n",
      "INFO:tensorflow:loss = 0.49489176, step = 1402 (5.029 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1428 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 38.5992\n",
      "INFO:tensorflow:Saving checkpoints for 1530 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.4189\n",
      "INFO:tensorflow:loss = 0.56307954, step = 1602 (5.027 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1632 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.5548\n",
      "INFO:tensorflow:Saving checkpoints for 1734 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.904\n",
      "INFO:tensorflow:loss = 0.49017936, step = 1802 (4.923 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1836 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 40.8268\n",
      "INFO:tensorflow:Saving checkpoints for 1938 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 41.1172\n",
      "INFO:tensorflow:loss = 0.4937549, step = 2002 (4.948 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2002 into models/model_WIDE_AND_DEEP_1559813118/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.4937549.\n",
      "fit done\n",
      "CPU times: user 2min 45s, sys: 6.61 s, total: 2min 52s\n",
      "Wall time: 1min 34s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# This can be found with\n",
    "# wc -l train.csv\n",
    "train_sample_size = 800000\n",
    "train_steps = train_sample_size/BATCH_SIZE # 8000/40 = 200\n",
    "\n",
    "m.fit(input_fn=generate_input_fn(train_file, BATCH_SIZE), steps=train_steps)\n",
    "\n",
    "print('fit done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate the accuracy of the model\n",
    "Let's see how the model did. We will evaluate all the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-06-06-09:31:47\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1559813118/model.ckpt-2002\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [50/500]\n",
      "INFO:tensorflow:Evaluation [100/500]\n",
      "INFO:tensorflow:Evaluation [150/500]\n",
      "INFO:tensorflow:Evaluation [200/500]\n",
      "INFO:tensorflow:Evaluation [250/500]\n",
      "INFO:tensorflow:Evaluation [300/500]\n",
      "INFO:tensorflow:Evaluation [350/500]\n",
      "INFO:tensorflow:Evaluation [400/500]\n",
      "INFO:tensorflow:Evaluation [450/500]\n",
      "INFO:tensorflow:Evaluation [500/500]\n",
      "INFO:tensorflow:Finished evaluation at 2019-06-06-09:32:12\n",
      "INFO:tensorflow:Saving dict for global step 2002: accuracy = 0.767205, accuracy/baseline_label_mean = 0.251165, accuracy/threshold_0.500000_mean = 0.767205, auc = 0.7288591, auc_precision_recall = 0.4861321, global_step = 2002, labels/actual_label_mean = 0.251165, labels/prediction_mean = 0.25349167, loss = 0.49896336, precision/positive_threshold_0.500000_mean = 0.6459558, recall/positive_threshold_0.500000_mean = 0.1618458\n",
      "evaluate done\n",
      "Accuracy: 0.767205\n",
      "{'loss': 0.49896336, 'accuracy': 0.767205, 'labels/prediction_mean': 0.25349167, 'labels/actual_label_mean': 0.251165, 'accuracy/baseline_label_mean': 0.251165, 'auc': 0.7288591, 'auc_precision_recall': 0.4861321, 'accuracy/threshold_0.500000_mean': 0.767205, 'precision/positive_threshold_0.500000_mean': 0.6459558, 'recall/positive_threshold_0.500000_mean': 0.1618458, 'global_step': 2002}\n",
      "CPU times: user 1min 5s, sys: 2.27 s, total: 1min 7s\n",
      "Wall time: 30.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "eval_sample_size = 200000 # this can be found with a 'wc -l eval.csv'\n",
    "eval_steps = eval_sample_size/BATCH_SIZE # 2000/40 = 50\n",
    "\n",
    "results = m.evaluate(input_fn=generate_input_fn(eval_file), \n",
    "                     steps=eval_steps)\n",
    "print('evaluate done')\n",
    "\n",
    "print('Accuracy: %s' % results['accuracy'])\n",
    "print(results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model directory = models/model_WIDE_1559813556\n",
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1a46f83fd0>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_WIDE_1559813556'}\n",
      "estimator built\n"
     ]
    }
   ],
   "source": [
    "MODEL_TYPE2 = 'WIDE'\n",
    "model_dir = create_model_dir(model_type=MODEL_TYPE2)\n",
    "m1 = get_model(model_type=MODEL_TYPE2, model_dir=model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into models/model_WIDE_1559813556/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.6931466, step = 1\n",
      "INFO:tensorflow:global_step/sec: 20.8531\n",
      "INFO:tensorflow:loss = 0.45659462, step = 101 (4.797 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.9448\n",
      "INFO:tensorflow:loss = 0.5334404, step = 201 (1.726 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.8021\n",
      "INFO:tensorflow:loss = 0.47971246, step = 301 (1.730 sec)\n",
      "INFO:tensorflow:global_step/sec: 56.9089\n",
      "INFO:tensorflow:loss = 0.5206535, step = 401 (1.757 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.877\n",
      "INFO:tensorflow:loss = 0.5690068, step = 501 (1.728 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.6944\n",
      "INFO:tensorflow:loss = 0.46700844, step = 601 (1.733 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.3796\n",
      "INFO:tensorflow:loss = 0.5033304, step = 701 (1.743 sec)\n",
      "INFO:tensorflow:global_step/sec: 58.0343\n",
      "INFO:tensorflow:loss = 0.56871605, step = 801 (1.723 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.7041\n",
      "INFO:tensorflow:loss = 0.4968151, step = 901 (1.733 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.5971\n",
      "INFO:tensorflow:loss = 0.49897647, step = 1001 (1.736 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.3155\n",
      "INFO:tensorflow:loss = 0.5733128, step = 1101 (1.745 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.9594\n",
      "INFO:tensorflow:loss = 0.5235764, step = 1201 (1.725 sec)\n",
      "INFO:tensorflow:global_step/sec: 56.7936\n",
      "INFO:tensorflow:loss = 0.46780455, step = 1301 (1.761 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.7391\n",
      "INFO:tensorflow:loss = 0.44247615, step = 1401 (1.732 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.6913\n",
      "INFO:tensorflow:loss = 0.5041066, step = 1501 (1.733 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.2479\n",
      "INFO:tensorflow:loss = 0.47314012, step = 1601 (1.747 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.5307\n",
      "INFO:tensorflow:loss = 0.5302299, step = 1701 (1.738 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.9756\n",
      "INFO:tensorflow:loss = 0.5448985, step = 1801 (1.725 sec)\n",
      "INFO:tensorflow:global_step/sec: 57.0629\n",
      "INFO:tensorflow:loss = 0.49571082, step = 1901 (1.752 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into models/model_WIDE_1559813556/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.5170178.\n",
      "fit done\n"
     ]
    }
   ],
   "source": [
    "train_sample_size = 800000\n",
    "train_steps = train_sample_size/BATCH_SIZE # 8000/40 = 200\n",
    "m1.fit(input_fn=generate_input_fn(train_file, BATCH_SIZE), steps=train_steps)\n",
    "\n",
    "print('fit done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-06-06-09:34:21\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_1559813556/model.ckpt-2000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [50/500]\n",
      "INFO:tensorflow:Evaluation [100/500]\n",
      "INFO:tensorflow:Evaluation [150/500]\n",
      "INFO:tensorflow:Evaluation [200/500]\n",
      "INFO:tensorflow:Evaluation [250/500]\n",
      "INFO:tensorflow:Evaluation [300/500]\n",
      "INFO:tensorflow:Evaluation [350/500]\n",
      "INFO:tensorflow:Evaluation [400/500]\n",
      "INFO:tensorflow:Evaluation [450/500]\n",
      "INFO:tensorflow:Evaluation [500/500]\n",
      "INFO:tensorflow:Finished evaluation at 2019-06-06-09:34:39\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.766125, accuracy/baseline_label_mean = 0.251165, accuracy/threshold_0.500000_mean = 0.766125, auc = 0.7227874, auc_precision_recall = 0.4757249, global_step = 2000, labels/actual_label_mean = 0.251165, labels/prediction_mean = 0.26990917, loss = 0.5024433, precision/positive_threshold_0.500000_mean = 0.6087969, recall/positive_threshold_0.500000_mean = 0.19260247\n",
      "evaluate done\n",
      "Accuracy: 0.766125\n",
      "{'loss': 0.5024433, 'accuracy': 0.766125, 'labels/prediction_mean': 0.26990917, 'labels/actual_label_mean': 0.251165, 'accuracy/baseline_label_mean': 0.251165, 'auc': 0.7227874, 'auc_precision_recall': 0.4757249, 'accuracy/threshold_0.500000_mean': 0.766125, 'precision/positive_threshold_0.500000_mean': 0.6087969, 'recall/positive_threshold_0.500000_mean': 0.19260247, 'global_step': 2000}\n",
      "CPU times: user 44.3 s, sys: 1.56 s, total: 45.9 s\n",
      "Wall time: 21 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "eval_sample_size = 200000 # this can be found with a 'wc -l eval.csv'\n",
    "eval_steps = eval_sample_size/BATCH_SIZE # 2000/40 = 50\n",
    "\n",
    "results1 = m1.evaluate(input_fn=generate_input_fn(eval_file), \n",
    "                     steps=eval_steps)\n",
    "print('evaluate done')\n",
    "\n",
    "print('Accuracy: %s' % results1['accuracy'])\n",
    "print(results1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model directory = models/model_DEEP_1559813690\n",
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x1a53245a90>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_device_fn': None, '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_DEEP_1559813690'}\n",
      "estimator built\n"
     ]
    }
   ],
   "source": [
    "MODEL_TYPE3 = 'DEEP'\n",
    "model_dir = create_model_dir(model_type=MODEL_TYPE3)\n",
    "m2 = get_model(model_type=MODEL_TYPE3, model_dir=model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into models/model_DEEP_1559813690/model.ckpt.\n",
      "INFO:tensorflow:loss = 691.9402, step = 1\n",
      "INFO:tensorflow:global_step/sec: 13.2336\n",
      "INFO:tensorflow:loss = 0.5466021, step = 101 (7.559 sec)\n",
      "INFO:tensorflow:global_step/sec: 34.4949\n",
      "INFO:tensorflow:loss = 0.5623397, step = 201 (2.898 sec)\n",
      "INFO:tensorflow:global_step/sec: 27.6305\n",
      "INFO:tensorflow:loss = 0.5410743, step = 301 (3.619 sec)\n",
      "INFO:tensorflow:global_step/sec: 32.0376\n",
      "INFO:tensorflow:loss = 0.57229424, step = 401 (3.121 sec)\n",
      "INFO:tensorflow:global_step/sec: 32.0478\n",
      "INFO:tensorflow:loss = 0.61574495, step = 501 (3.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 29.0789\n",
      "INFO:tensorflow:loss = 0.5263978, step = 601 (3.439 sec)\n",
      "INFO:tensorflow:global_step/sec: 28.7083\n",
      "INFO:tensorflow:loss = 0.55579495, step = 701 (3.483 sec)\n",
      "INFO:tensorflow:global_step/sec: 33.7142\n",
      "INFO:tensorflow:loss = 0.6038082, step = 801 (2.968 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.5734\n",
      "INFO:tensorflow:loss = 0.5705882, step = 901 (2.809 sec)\n",
      "INFO:tensorflow:global_step/sec: 32.0727\n",
      "INFO:tensorflow:loss = 0.5521241, step = 1001 (3.118 sec)\n",
      "INFO:tensorflow:global_step/sec: 34.1533\n",
      "INFO:tensorflow:loss = 0.61906695, step = 1101 (2.928 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.7119\n",
      "INFO:tensorflow:loss = 0.57343984, step = 1201 (2.800 sec)\n",
      "INFO:tensorflow:global_step/sec: 36.3855\n",
      "INFO:tensorflow:loss = 0.5166308, step = 1301 (2.748 sec)\n",
      "INFO:tensorflow:global_step/sec: 31.9567\n",
      "INFO:tensorflow:loss = 0.5027303, step = 1401 (3.129 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.8717\n",
      "INFO:tensorflow:loss = 0.56246614, step = 1501 (2.788 sec)\n",
      "INFO:tensorflow:global_step/sec: 35.6776\n",
      "INFO:tensorflow:loss = 0.4970348, step = 1601 (2.803 sec)\n",
      "INFO:tensorflow:global_step/sec: 30.5814\n",
      "INFO:tensorflow:loss = 0.5545795, step = 1701 (3.270 sec)\n",
      "INFO:tensorflow:global_step/sec: 25.2145\n",
      "INFO:tensorflow:loss = 0.56509966, step = 1801 (3.966 sec)\n",
      "INFO:tensorflow:global_step/sec: 26.046\n",
      "INFO:tensorflow:loss = 0.52552336, step = 1901 (3.839 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into models/model_DEEP_1559813690/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.56686914.\n",
      "fit done\n"
     ]
    }
   ],
   "source": [
    "train_sample_size = 800000\n",
    "train_steps = train_sample_size/BATCH_SIZE # 8000/40 = 200\n",
    "m2.fit(input_fn=generate_input_fn(train_file, BATCH_SIZE), steps=train_steps)\n",
    "\n",
    "print('fit done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to \"careful_interpolation\" instead.\n",
      "INFO:tensorflow:Starting evaluation at 2019-06-06-09:36:55\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/model_DEEP_1559813690/model.ckpt-2000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Evaluation [50/500]\n",
      "INFO:tensorflow:Evaluation [100/500]\n",
      "INFO:tensorflow:Evaluation [150/500]\n",
      "INFO:tensorflow:Evaluation [200/500]\n",
      "INFO:tensorflow:Evaluation [250/500]\n",
      "INFO:tensorflow:Evaluation [300/500]\n",
      "INFO:tensorflow:Evaluation [350/500]\n",
      "INFO:tensorflow:Evaluation [400/500]\n",
      "INFO:tensorflow:Evaluation [450/500]\n",
      "INFO:tensorflow:Evaluation [500/500]\n",
      "INFO:tensorflow:Finished evaluation at 2019-06-06-09:37:17\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.752595, accuracy/baseline_label_mean = 0.251165, accuracy/threshold_0.500000_mean = 0.752595, auc = 0.6443728, auc_precision_recall = 0.38376752, global_step = 2000, labels/actual_label_mean = 0.251165, labels/prediction_mean = 0.25057766, loss = 0.544895, precision/positive_threshold_0.500000_mean = 0.57730263, recall/positive_threshold_0.500000_mean = 0.05589951\n",
      "evaluate done\n",
      "Accuracy: 0.752595\n",
      "{'loss': 0.544895, 'accuracy': 0.752595, 'labels/prediction_mean': 0.25057766, 'labels/actual_label_mean': 0.251165, 'accuracy/baseline_label_mean': 0.251165, 'auc': 0.6443728, 'auc_precision_recall': 0.38376752, 'accuracy/threshold_0.500000_mean': 0.752595, 'precision/positive_threshold_0.500000_mean': 0.57730263, 'recall/positive_threshold_0.500000_mean': 0.05589951, 'global_step': 2000}\n",
      "CPU times: user 52.3 s, sys: 1.79 s, total: 54.1 s\n",
      "Wall time: 25.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import numpy as np\n",
    "\n",
    "eval_sample_size = 200000 # this can be found with a 'wc -l eval.csv'\n",
    "eval_steps = eval_sample_size/BATCH_SIZE # 2000/40 = 50\n",
    "\n",
    "results2 = m2.evaluate(input_fn=generate_input_fn(eval_file), \n",
    "                     steps=eval_steps)\n",
    "print('evaluate done')\n",
    "\n",
    "print('Accuracy: %s' % results2['accuracy'])\n",
    "print(results2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由以上结果可知：\n",
    "WIDE_AND_DEEP，'loss': 0.49896336, 'accuracy': 0.767205\n",
    "WIDE，'loss': 0.5024433, 'accuracy': 0.766125\n",
    "DEEP，'loss': 0.544895, 'accuracy': 0.752595\n",
    "其中WIDE_AND_DEEP效果是最好的，原因也是显而易见的，它更加全面而且有针对性的训练了数据。\n",
    "\n",
    "PS：如何调用logloss作为评估参数，我一直没弄出来，希望老师解答一下，谢谢\n",
    "直播课同学的方法我用了，但是感觉不对，因为做出了完全相反的结果，他是直接“loss”取了负log。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "In this Juypter notebook, we have configured, created, and evaluated a Wide & Deep machine learning model, that combines the powers of a Linear Classifier with a Deep Neural Network, using TensorFlow's Estimator and Experiment classes.\n",
    "\n",
    "With this working example in your toolbelt, you are ready to explore the wide (and deep) world of machine learning with TensorFlow! Some ideas to help you get going:\n",
    "* Change the features we used today. Which columns do you think are correlated and should be crossed? Which ones do you think are just adding noise and could be removed to clean up the model?\n",
    "* Swap in an entirely new dataset! There are many datasets available on the web, or use a dataset you possess! Check out https://archive.ics.uci.edu/ml to find your own dataset. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
